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Uncalibrated  Dynamic  Visual  Servoing 

J.  A.  Piepmeier,  G.V.  McMurray,  H.  Lipkin 


Abstract — A  dynamic  quasi-Newton  method  for  uncalibrated, 
vision-guided  robotic  tracking  control  with  fixed  imaging  is  devel¬ 
oped  and  demonstrated.  This  method  does  not  require  calibrated 
kinematic  and  camera  models.  Robotic  control  is  achieved  at  each 
step  through  minimizing  a  nonlinear  objective  function  by  taking 
quasi-Newton  steps  and  estimating  the  composite  Jacobian  at 
each  step.  The  Jacobian  is  estimated  using  a  dynamic  recursive 
least  squares  algorithm.  Experimental  results  demonstrate  the 
validity  of  this  approach. 

Index  Terms — Uncalibrated  visual  servoing,  dynamic  nonlin¬ 
ear  least  squares,  Jacobian  estimation. 

I.  Introduction 

This  article  develops  an  uncalibrated,  vision-guided,  robotic 
control  method  with  a  fixed  imaging  system.  The  controller 
is  a  dynamic  quasi-Newton  method  based  on  nonlinear  least 
squares  optimization  methods.  The  system  model  is  approx¬ 
imated  using  a  dynamic  Jacobian  estimation  scheme.  Using 
this  controller,  the  robot  can  be  servoed  to  both  static  and 
moving  targets,  even  with  uncalibrated  robot  kinematics  and 
camera  models.  The  control  method  is  completely  independent 
of  robot  type,  camera  type,  and  camera  location.  In  other 
words,  it  is  independent  of  the  system  model. 

By  rejecting  a  model  based  paradigm,  the  control  algorithm 
eliminates  the  necessity  of  extensive  system  modeling,  sys¬ 
tem  re -calibration,  and  ancillary  hardware  that  constrains  the 
workspace.  In  a  manufacturing  setting,  these  are  nonvalue- 
added  activities;  reducing  or  eliminating  the  amount  of  time 
spent  on  them  reduces  the  overall  cost  of  a  product. 

Specifically,  the  contributions  of  this  work  are: 

•  Algorithms  are  explicitly  derived  to  track  a  moving 
target  using  uncalibrated  visual  servoing  with  stationary 
cameras. 

•  A  theoretical  basis  for  uncalibrated  vision-guided  control 
algorithms  is  developed  based  on  the  direct  minimization 
of  Frobenius  norms. 

•  The  developed  algorithms  are  experimentally  verified  for 
uncalibrated  vision-guided  robotic  control  and  tracking. 

Quasi-Newton  methods  implementing  Jacobian  estimation 
have  been  used  with  success  to  servo  a  robot  to  a  static  target. 
This  work  seeks  to  thoroughly  develop  the  quasi-Newton 
control  method  for  target  tracking. 

There  are  several  instances  in  the  literature  where  Jacobian 
estimation  is  used  in  visual  servoing.  Hosoda  and  Asada  [1] 
and  Hosoda,  Igarashi,  and  Asada  [2]  have  presented  uncal¬ 
ibrated  vision-guided  robotic  control  for  static  targets  using 
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fixed  cameras..  The  Jacobian  estimation  scheme  utilized  in  [1] 
is  similar  to  an  exponentially  weighted  recursive  least  squares 
update  equation  for  vectors.  However,  the  vector  format  is 
applied  to  the  Jacobian  matrix  estimation  problem.  In  [3], 
Asada,  Tanaka,  and  Hosoda  extend  the  Jacobian  estimation 
scheme  to  eye-in-hand  stereo  tracking  of  moving  objects  using 
static  reference  points  to  estimate  target  motion. 

Jagersand  [4],  [5]  takes  the  approach  of  a  nonlinear  least 
squares  optimization  method  utilizing  a  trust  region  method 
and  Broyden  estimation.  The  moving  target  scenario  is  not 
addressed. 

Previous  work  has  shown  the  feasibility  of  uncalibrated 
visual  servo  control  for  stationary  targets.  However,  a  control 
law  for  the  moving  target  scenario  has  not  been  rigorously 
developed. 

II.  An  Uncalibrated  Dynamic  Visual  Servoing 
Method 

A  stationary  vision  system  is  assumed  that  can  sense 
sufficient  end-effector  and  target  features  to  locate  both  bodies 
in  space.  This  renders  the  target  features,  y*  it),  as  functions 
of  only  time  t  and  the  end-effector  features,  y  (9),  as  functions 
of  only  the  robot  joint  angles,  6  £  Rn.  It  is  important  to  note 
that  t  and  6  are  independent  variables  since  as  time  varies 
the  joint  angles  can  be  held  constant,  and  conversely  at  every 
given  time  the  joint  angles  can  take  on  any  values.  There 
are  no  assumptions  yet  about  target  tracking.  To  optimally 
track  the  target,  a  constraint  relationship  is  imposed  between 
6  and  t  so  joint  angles  are  selected  as  a  function  of  time, 
6  (t)  =  g  (y*  (t)).  This  establishes  an  optimal  end-effector 
trajectory  y  {9{t))  to  follow  the  moving  target.  The  constraint 
is  established  by  minimizing  the  tracking  error,  /  £  Ft as 
seen  in  the  image  plane 

f{9,t)  =  y{9)-y*{t)  (1) 

The  combined  transformations  of  forward  kinematics  and 
imaging  geometry  render  f  (9.  t)  a  highly  nonlinear  function. 
This  multivariate  optimization  problem  is  solved  at  each 
increment  by  a  dynamic  quasi-Newton  controller  (Section  II- 
A)  with  a  dynamic  Jacobian  estimator  (Section  II-B). 

A.  A  Dynamic  Newton ’s  Method 

The  imposed  trajectory  6  ( t )  that  causes  the  end-effector  to 
follow  the  target  is  established  by  minimizing  the  image  error 
squared 

F(9,t)  =  ±fT(9,t)mt) 

which  can  also  be  modified  by  a  weighting  matrix  but  is 
omitted  for  simplicity.  The  Taylor  series  expansion  about  (9,  t) 
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is 


F(0  +  hg,t  +  hf)  —  F  (0,t)  +  Fghg  -f  Fthi  +  ... 


where  Fg  and  Ft  are  partial  derivatives  and  hg  and  /it  are 
increments  of  6  and  t.  For  a  fixed  sampling  period  ht,  F  is 
minimized  by  solving 

_  dF(6  +  hg ,  t.  +  ht ) 

“  86 

=  Fg  +  Fgghg  +  Ftght  +  O  (/r)  (2) 


where  O  ( h 2)  indicates  second  order  terms  in  ht  and  hg. 
Dropping  these  terms,  rearranging,  expanding  the  partials,  and 
adding  6  gives  what  is  referred  to  here  as  a  dynamic  Newton’s 
method 

e  +  hg=e-{jTj  +  s)~1jT(f  +  ^ht)  (3) 


where  J  is  the  joint-to-image  feature  error  composite  Jacobian 
and 


j  =  ^l  S  =  —f 
'  ~  d0 '  80  1 

Fg  =  JTf ,  Fgg  =  JT  J  +  S,  Ftg  =  J 


d£ 

d  t 


To  compute  the  terms  S  and  J  analytically  requires  a 
calibrated  system  model.  The  term  S  is  difficult  to  estimate, 
but  as  0  approaches  the  solution,  it  approaches  zero  and  it 
is  often  dropped  to  give  what  is  sometimes  called  a  Gauss- 
Newton  method.  It  can  be  shown  that  for  a  small  enough 
time  increment,  ht,  the  method  is  well-defined  for  all  0  and 
converges  linearly  to  a  finite  steady-state  error  [6].  When 
an  estimated  Jacobian,  J,  is  used  the  algorithm  becomes  a 
dynamic  quasi-(Gauss-)Newton  method  such  that  at  the  kth 
increment. 


0k+1  =0k-  ( Jj Jk)-1  Jl (fk  +  -g ht )  (4) 

where  hg  =  0k+1  -  0k. 

The  qualifier  “dynamic”  specifically  refers  to  the  presence 
of  the  error  velocity  term  g-  which  is  used  to  linearly 
predict  the  error  vector  at  the  next  time  increment  as  fk+ 1 
fk  +  7 {r/it,  assuming  the  robot  remains  at  its  current  position. 
Clearly  more  complex  predictors  are  possible  but  the  linear 
one  has  appealing  simplicity.  When  the  predictor  is  absent  the 
method  is  referred  to  here  as  “static”  so  (4)  becomes 


0k+i=0k-{JlJk)-\jlfk  (5) 


which  is  the  basis  for  previous  work  by  [1],  [5],  and  [7].  The 
derivation  is  similar  to  the  dynamic  case  if  the  error  is  only  a 
function  of  the  joint  angles,  f{0). 

The  static  and  dynamic  methods  coincide  in  two  distinct 
cases.  First,  if  the  time  step  ht  is  made  sufficiently  small  in 
relation  to  g-  then  it  appears  as  if  the  error  j),  is  constant 
or  static  (sometimes  this  is  called  a  quasi-static  condition). 
This  suggests  that  the  dynamic  method  should  be  expected  to 
track  with  slower  required  sampling  rates  and  to  track  more 
demanding  error  functions  compared  to  the  static  method. 
Second,  since  =  gy  (0)  -  y*  (t))  =  -g,  then 
vanishes  if  the  target  is  fixed,  y*  =  0.  Thus  the  predictor 
term  attempts  to  compensate  for  target  motion  should  result 
in  better  ttacking  than  the  static  method  for  moving  targets. 


B.  A  Dynamic  Quasi-Newton  Method 

As  with  Newton’s  method,  the  moving  target  scenario  re¬ 
quires  that  the  appropriate  derivatives  are  included  in  estimat¬ 
ing  the  Jacobian,  .Jk.  For  static  target  servoing,  Jagersand  in 
[4]  and  [5]  employs  Broyden’s  method  for  Jacobian  estimation. 
In  this  section,  a  dynamic  Broyden’s  method  is  derived.  Next, 
a  similar  Jacobian  estimation  scheme  is  developed  that  uses 
a  recursive  least  squares  algorithm  and  is  more  stable  in 
the  presence  of  noise.  Finally,  the  steady  state  performance 
of  a  quasi-Newton  method  using  the  recursive  least  squares 
estimation  is  discussed. 

1 )  Derivation  of  a  Dynamic  Jacobian  Estimation  Scheme: 
Several  Jacobian  estimation  schemes  using  Broyden’s  method 
or  a  similar  variant  have  been  implemented  as  discussed  in 
[1]-  [5],  Analogous  to  the  controller,  the  proposed  dynamic 
Broyden  update  contains  an  additional  term,  g-ht.  This 
dynamic  Broyden’s  update  is  derived  for  a  moving  target 
scenario  by  extending  the  derivation  of  the  static  Broyden’s 
update  given  in  Dennis  and  Schnabel  [8]. 

The  affine  model  (a  linear  model  that  does  not  necessarily 
pass  through  the  origin  [8])  of  the  error  function  f(0,t)  is 
a  first  order  Taylor  series  approximation  denoted  as  m(0,t). 
The  model  is  updated  at  each  iteration  and  for  the  k 1,1 , 

mk(0,t)  =  fk  +  Jk(0  -  0k)  +  ^  (t  -  tk )  (6) 

Requiring  that  the  kth  affine  model  correctly  specifies  the 
error  at  the  (k  —  1  )th  increment,  mk{0k-i,tk-i)  =  fk- 1  = 
(/ {0k-i ,  tk-i)),  yields  the  so-called  secant  equation 

Jkhg  +  ^ ht  =  A  /  (7) 

dt 

where  A /  =  fk  —  fk-\.  Broyden’s  method  requires  that  (7) 
holds.  Subtracting  Jk_\hg  from  each  side,  rearranging,  and 
transposing  gives, 

hTg\JT  =  (Af  -  ^ ht  -  Jk-lhg)T  (8) 


where  A  J  =  Jk—Jk_i .  The  Jacobian  update  A  J  is  selected  to 
minimize  the  Frobenius  norm  ||A  J\\p  =  (^(AJ)^) s  subject 
to  the  constraint  (8)  where  (A  J),j  indexes  AJ.  By  stacking 
the  elements  into  a  vector  and  rewriting  (8)  accordingly,  the 
problem  is  cast  into  a  familiar  form  with  a  minimum  norm 
solution.  Unstacking  the  result  gives  the  dynamic  Broyden 
update  (see  [6],  [9]  for  details). 


Jk  —  Jk— i  T 


(Af  -  Jk.jhg  -  ght)h 

hghg 


(9) 


The  qualifier  “dynamic”  specifically  refers  to  the  presence 
of  the  error  velocity  term  g-.  If  it  is  assumed  that  the  affine 
model  is  only  a  function  of  the  joint  angles,  to  (0),  then  the 
“static”  Broyden  update  results. 


Jk 


Jk- 1  + 


(A  f-Jk-thg)^ 

hg  hg 


(10) 


The  static  and  dynamic  Broyden  methods  are  entirely  analo¬ 
gous  to  the  static  and  dynamic  Newton  methods.  They  coincide 
when  either  the  time  increment  is  very  small  or  the  target  is 
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stationary.  The  dynamic  method  should  be  expected  to  be  more 
effective  with  slower  sampling  rates  and  should  have  better 
tracking  of  moving  targets. 

As  a  consequence  of  the  secant  condition  the  affine  models 
are  piecewise  continuous  so  at  the  ( k  —  l)th  increment, 

m-k  (0k-i,tk-i)  ~  mu- 1  (0k-i,tk-i)  =  0 


which  is  easily  verified  using  (6)  for  mu  and  mk- i  at 
and  (7).  The  kth  affine  model  must  contain  the 
points  fk-i  and  fu  as  well  as  satisfy  the  secant  equation. 
However  consider  the  case  where  noise  is  present  in  the 
system.  The  measurement  fk  includes  a  nominal  value  of 
the  eiror  function  fk  as  well  as  system  noise  Snoise  so  that 
fk  =  fk  +  Snoise ■  In  the  event  of  large  values  for  Snoise, 
requiring  that  the  secant  equation  holds  may  result  in  an  affine 
model  mu  which  introduces  large  errors  in  Jk-  Large  errors  in 
J),  will  result  in  poor  tracking.  Thus,  the  dynamic  Broyden’s 
method  may  perform  poorly  for  noisy  systems  and  can  be 
improved  by  a  recursive  formulation  that  provides  a  filtering 
action. 

2)  A  Dynamic  Jacobian  Estimation:  Recursive  Least 
Squares:  Increased  stability  can  be  achieved  using  an  ex¬ 
ponentially  weighted  recursive  least  squares  (RLS)  algorithm 
[10]  that  minimizes  a  cost  function  G  k  based  on  the  change 
in  the  affine  model 


k 

Gk  =  ^  Afc_!  II mk  (0i- i)  -  m*_  1  (0i  i)  J  (11) 

i= 1 


The  objective  function  in  (11)  is  an  exponentially  weighted 
sum  of  the  differences  between  the  current  affine  model 
and  past  affine  models  where  0  <  A  <  1.  By  minimiz¬ 
ing  Gk  the  secant  equation  (7)  no  longer  holds  nor  are 
the  affine  models  piecewise  continuous.  Minimizing  Gk  is 
equivalent  to  minimizing  the  Frobenius  norm  of  the  term 
(AM)1  A  (AM)  where  AM  is  a  k  x  m  matrix  whose  ith 
column  is  mk  (9i-i,  L-i )  —  m,;_i  i ,  t,;_i )  and  A  is  a  k  x  k 
diagonal  matrix  with  \k~l  at  the  ith  diagonal  element. 

Following  the  approach  taken  in  Brogan  [9],  it  can  be  shown 
[6]  that  the  desired  recursive  estimation  scheme  that  minimizes 
(11)  is  given  by. 


Jk 


Pk 


Jk- 1  + 


(( fk  ~  fk- 1)  ~  Jk-ihg  -  §|/it)  hfPk-! 


A  +  hi  Pk—ihg 


(12) 


f  p  _  Pk-ihghl Pk~i  A 

A  V  fc_1  A  +  hT0Pk-lhg) 


(13) 


As  before,  the  static  version  omits  the  term  ^fhi- 

The  form  is  similar  to  the  dynamic  Broyden’s  method  in 
(9)  with  the  exception  of  A  and  Pk  .  The  weighting  parameter 
0  <  A  <  1  can  be  tuned  to  average  in  more  or  less  of  the 
previous  information  where  the  effective  number  of  terms  is 
estimated  by  j-zj.  As  shown  in  [9],  it  is  possible  to  use  more 
complex  weighting  matrices  in  the  formulation.  However  this 
has  not  been  done  here  due  to  a  lack  of  a  specific  physical 
justification  and  the  desire  for  the  simplest  equations. 


The  RLS  algorithm  is  often  used  for  system  identification 
[11],  Indeed,  the  formulation  presented  here  identifies  the 
system  model  (the  Jacobian)  for  visually  guided  control.  It 
is  interesting  to  note  that  in  the  literature  [9],  [10]  the  RLS 
algorithm  is  typically  used  for  the  estimation  of  a  vector; 
however,  the  algorithm  can  be  shown  [6]  to  extend  to  the 
estimation  of  a  matrix.  Equations  (12)  and  (13)  define  a 
recursive  update  for  estimating  J*,. 

A  dynamic  quasi-Newton  method  using  the  RLS  Jacobian 
estimation  scheme  follows: 

Algorithm  1:  Given  /  :  Rn  ->■  Rm;90,  0!  G  Rn;J0  G 
Rmxn,P0  G  />”' *  " .  A  G  (0,1) 

Do  for  k  =  1,2, ... 

A/  =  fk  -  fk- i 

~dt^t  =  ~(y*k  —  vt- 1) 

he  =  9k  —  9k- i 

Jk  =  Jk- 1  +  (A  +  hi  Pk-ihg) 

•(A  f-jk-ihg-2£-ht)hJPk-i 

Pk  =  j(Pk- 1  ~  (A  +  hi  Pk-ihg)  (Pk-ihehJ  Pk~i)) 

0k  n  =0k-  (JlJk^J^ifk  +  Zfeht) 

End  for 

End 


C.  Convergence  Analysis  of  Dynamic  Quasi-Newton 
Method  using  RLS  Jacobian  Estimation 

The  previous  section  gives  a  clear  rational  for  the  formula¬ 
tion  of  the  Jacobian  estimation  scheme  as  an  RLS  algorithm. 
In  this  section,  the  convergence  properties  using  Algorithm  1 
are  addressed.  Let  9'j,  represent  the  joint  position  for  which 
the  objective  function  is  minimized.  For  the  dynamic  quasi- 
Newton  method  using  RLS  Jacobian  estimation,  it  is  shown 
that  under  certain  assumptions,  the  norm  of  the  joint  error 
9*k  —  9k  is  bounded.  This  is  accomplished  using  the  steady 
state  properties  of  the  RLS  algorithm  given  by  (12)  and  (13) 
as  discussed  in  [12], 

It  is  readily  verified  that  the  dynamic  quasi-Newton  method 
(4)  can  be  expressed  as  the  equivalent  form 

hk+1  =  JUfk  +  ^ht  +  J*kh*ek  +  Ekh*0h  +  JkA9*)  (14) 

by  using  the  substitutions  h*0k  =  9*k  -9k.  J^  =  (Jj.  Jk)  , 
J*  =  Jk{9%),  Ek  =  Jk-J*k*A9*  =  9*k+1  —9*k  and  the  identity 
Jf  Jk  =  I.  From  the  Taylor  series  expansion  of  9*k+1 

—jj^ht  =  9l+1  -9k+  0{h\) 

and  since  ^f-ht  =  ~^jfht  =  -Jk  d9^t  1  ht  then 

d-^ht  =  -J*kA9*  -  JkO  (hj)  (15) 

so  that  with  fk  =  fk  (9kQk)  =  0  equation  (14)  becomes 

h*9k+1  =  JUfk  ~  fk  +  J*khk  +  EkKh  +  Ek A9*  -  J*kO(h;)) 

(16) 
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If  J*  is  Lipschitz  continuous  with  a  Lipschitz  constant  7,  then 
taking  the  norm  of  (16)  results  in 

l!^+1ll<ll-A+ll(|ll^JI  +  \\mmj\ 

+  ||j+||||^||||Ar||  +  Glo(/li2) 

where  c\  is  some  constant.  Recall  that  J jJ. 
Assuming  that  ||(jJj*)_1||  <  where  k  is  the  smallest 
eigenvalue  of  Jj  J*,  and  ||jj||  <  a  then 

+  II^II)I1^JI  07) 

rv  Z 

+  ^\\Ek\\\\Ae*\\+c,\\hi\\ 

Assume  that  ||/ig  ||  and  ||£^||  are  bounded  such  that, 

Cj^G;\\hlk\\  +  \\Ek\\)<a  (18) 

rv  Z 

^\\Ek\\\\A0*\\+Ci\\h2t\\<P  (19) 

K 

for  some  0  <  a  <  1  and  ft  >  0.  Then,  requiring  that  the  initial 
error  \\hgg\\  <  jZ-  ,  assures  that  the  sequence  hf  is  linearly 
convergent  and  bounded  for  all  k. 

As  discussed  in  [13]  and  [12],  since  the  bound  on  |j£^||  is 
based  on  A,  system  and  measurement  noise,  and  the  amount  of 
variation  in  J*,  it  is  difficult  to  rigorously  characterize  ||i?fc|] 
such  that  (18)  holds  for  a  <  1.  However,  it  is  plausible  that 
for  non-pathological  tracking  problems,  an  upper  bound  on 
|]f?^||  exists,  and  thus  the  quasi-Newton  method  is  convergent 
and  stable.  Experimental  results  verify  this  assumption,  and 
the  reader  is  refered  to  [6]  for  further  verification. 

D.  A  Comparison  of  Uncalibrated  Static  and  Dynamic  Con¬ 
trollers  and  Jacobian  Estimators 

A  convergence  analysis  for  the  static  controller  can  be 
done  in  a  similar  manner  to  Section  II-C.  However,  the 
requirement  that  ||^||  is  bounded  may  not  necessarily  be 
met  by  a  static  Jacobian  estimator.  Simulations  performed  in 
MATLAB  demonstrate  the  improved  tracking  and  stability  of 
the  dynamic  controller  with  dynamic  Jacobian  RLS  estimation 
(Algorithm  1)  over  the  static  controller  with  static  Jacobian 
RLS  estimation  (Algorithm  1  without  the  7=>F^  terms)  which 
exhibits  stability  problems  for  faster  moving  targets. 

The  experimental  workcell  (see  Section  III)  is  simulated 
using  Corke’s  Robotics  Toolbox  and  Machine  Vision  Toolbox 
[14],  The  first  three  joints  (two  revolute  and  one  prismatic) 
of  an  AdeptOne  robot  servo  the  toolpoint  to  follow  a  moving 
target  with  one  feature  point  on  each  tracked  by  two  cameras. 
The  target  moves  circularly  in  the  x-y  plane  and  sinusoidally  in 
the  z  direction.  Simulations  were  run  for  25  seconds  with  a  100 
ms  sampling  time  at  various  target  speeds.  Uniform  ±0.5  pixel 
noise  was  added  to  the  data.  Twenty-five  simulations  were  per¬ 
formed  at  each  speed  for  each  controller,  and  the  average  RMS 
tracking  error  in  pixels  is  plotted  for  both  cameras  in  Figure 
1.  At  slower  speeds,  the  visual  servoing  problem  is  quasi¬ 
static  and  the  two  controllers  behave  similarly  at  expected.  As 
the  target  motion  increases,  the  static  controller’s  performance 
degrades  with  the  tracking  lagging  substantially  and  often 
becoming  wildly  divergent.  Clearly,  the  static  controller  is 
unsuitable  for  tracking  moving  targets. 
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Fig.  1.  A  dynamic  controller  using  Algorithm  1,  and  a  static  controller 
(Algorithm  1  without  the  AfLfi  terms)  are  compared  at  increasing  target 
speeds  for  a  3DOF  tracking  problem.  The  static  controller  causes  the  robot 
to  lag  behind  the  target  motion  and  often  goes  unstable  for  higher  target 
speeds.  While  tracking  error  increases  at  higher  speeds,  the  dynamic  controller 
remains  stable. 


III.  Experimental  System 

In  [6]  an  initial  verification  of  the  dynamic  quasi-Newton 
method  and  dynamic  RLS  Jacobian  estimation  is  given  in 
using  a  small  reconfigurable  Robix^M  RCS-6  robot  (in  a  2- 
DOF  planar  configuration),  a  single  Pulnix  camera,  a  MuTech 
frame  grabber  and  two  personal  computers.  This  system 
demonstrated  stable  and  convergent  tracking  of  a  moving 
target. 

The  system  used  here  employs  stereo  vision  and  the  first 
three  degrees-of-freedom  on  an  AdeptOne  robot.  Image  pro¬ 
cessing  and  pattern  recognition  is  accomplished  on  a  400 
MHz  Pentium  II  running  Windows98  utilizing  the  MVS-8000 
software  including  the  PatQuick  tool  from  Cognex.  The  end 
effector  is  gripping  a  tool  that  holds  a  screw.  The  tool  has  been 
covered  with  a  small  pattern  easily  identifiable  by  the  Cognex 
software.  The  target  is  comprised  of  a  small  paper  pad  with 
a  pattern  drawn  on  it  moving  on  an  industrial  conveyor  belt. 
The  robot/vision  system  servos  to  the  centroid  of  the  pattern 
and  tracks  it  on  the  belt. 

Various  hardware  and  software  elements  perform:  image 
acquisition,  image  processing,  algorithmic  calculations,  and 
the  sending  of  control  commands.  During  one  control  cycle, 
two  images  are  captured;  the  second  one  is  used  to  estimate 
the  position  of  target  and  end  effector  at  the  end  of  the  control 
cycle,  because  it  is  algorithmically  important  to  sense  the  result 
of  the  most  recent  control  signal  before  calculating  the  next 
update.  Velocity  commands  are  sent  to  the  robot  controller, 
creating  a  smooth  motion. 

The  forgetting  factor  A  in  (12)  and  (13)  is  used  to  tune  the 
memory  of  the  Jacobian  estimation.  A  value  of  A  =  0.5  is  used 
until  convergence  is  achieved  and  then  a  value  of  A  =  0.99 
is  used.  This  creates  a  shorter  memory  while  the  system  is 
learning  the  Jacobian  and  a  longer  memory  during  steady  state 
tracking. 
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No  model  of  target  motion  is  used,  the  stereo  vision  system 
is  completely  uncalibrated,  and  none  of  the  robot’s  kinematic 
parameters  are  used  in  the  control  algorithm.  Due  to  hardware 
limitations,  the  update  rate  is  a  very  slow  0.352s  (2.8  Hz),  yet 
the  system  is  still  able  to  exhibit  convergence  and  steady  state 
tracking  of  a  moving  target. 

IV.  Experimental  Results 

The  initial  Jacobian  is  estimated  using  three  initial  moves 
by  each  of  the  three  joints  being  controlled  by  the  dynamic 
quasi-Newton  method.  The  target  is  kept  stationary  initially 
due  to  a  limited  field  of  view.  The  target  is  then  moved  by 
a  conveyor  belt  at  2.5cm/s  which  corresponds  to  an  average 
motion  of  26  pixels/s  and  12  pixels/s  for  Cameras  0  and  1 
respectively. 

Figures  2-4  give  experimental  image  feature  data  with  two 
image  captures  per  control  cycle.  Figure  2  shows  the  tracking 
error  norm  for  each  camera.  The  robot  converges  on  the 
target  smoothly,  and  the  error  converges  to  a  nominal  value. 
Convergence  is  achieved  in  8  control  cycles  after  which  the 
tracking  error  is  8.6  pixels  for  Camera  0  and  10.0  pixels  for 
Camera  1 .  Slight  breaks  in  the  tracking  error  indicate  a  failure 
of  the  vision  system  to  identify  target  or  robot  features.  In  the 
event  of  insufficient  data,  no  move  is  made. 

Figure  3  and  4  show  the  image  features  for  the  end  effector 
and  the  target  point  as  seen  by  Cameras  0  and  1.  Without 
known  camera  or  robot  models,  the  end-effector  tracks  the 
target  point.  The  tracking  does  display  a  slight  oscillatory 
nature.  This  could  be  the  result  of  the  memory  inherent  in 
the  recursive  least  squares  Jacobian  estimation. 

Figures  2-4  demonstrate  that  the  algorithm  is  able  to  con¬ 
verge  on  and  track  a  target  moving  on  a  conveyor  belt.  It  is  also 
interesting  to  examine  the  algorithm’s  response  to  an  abrupt 
change  in  the  target’s  motion.  Figure  5  shows  a  detail  of  the 
RMS  tracking  error  in  pixels  as  the  target  moves  back  and 
forth  in  the  camera’s  field  of  view.  During  this  experiment, 
the  target  is  moving  at  about  42  pixels/s  and  25  pixels/s  for 
Cameras  0  and  1  respectively.  This  corresponds  to  an  average 
movement  of  about  15  and  9  pixels  per  control  cycle  for  the 
respective  cameras.  The  arrows  in  the  figure  denote  the  points 
where  the  target  motion  was  changed  by  reversing  the  motion 
conveyor  belt.  As  would  be  expected,  the  algorithm  overshoots 
by  10-20  pixels,  but  regains  tracking  within  roughly  3  control 
cycles. 

The  cameras  are  positioned  such  that  one  pixel  corresponds 
to  roughly  to  1-2  mm.  For  the  tracking  of  a  steadily  moving 
target  shown  in  Figure  2,  the  average  image  errors  are  on  the 
order  of  10  or  less  pixels.  This  corresponds  to  a  Cartesian  error 
of  10-20  mm  or  less.  There  is  a  trade-off  in  selecting  a  camera 
position  for  this  type  of  a  system.  If  the  camera  is  far  away,  an 
error  of  5  pixels  results  in  a  greater  error  in  Cartesian  space 
than  a  5  pixel  error  when  the  camera  is  closely  positioned  to 
the  work  area.  However,  the  work  volume  becomes  constrained 
as  the  field  of  view  decreases. 

V.  Conclusions 

This  article  has  developed  and  demonstrated  a  dynamic 
quasi-Newton  method  for  visual  servo  control  of  uncalibrated 


Fig.  2.  Tracking  error  showing  convergence  after  8  control  cycles  (Two 
images  are  taken  per  control  cycle.)  After  8  control  cycles,  the  tracking  error 
is  8.6  pixels  for  Camera  0  and  10.0  pixels  for  Camera  1. 


Camera  0 


Fig.  3.  The  image  features  of  the  end  effector  and  the  target  as  seen  by 
Camera  0  demonstrate  the  convergent  tracking  of  a  moving  target.  The  average 
steady  state  tracking  error  is  10  pixels. 


robotic  systems  with  stationary  imaging.  Estimation  of  a 
Jacobian  representing  unknown  imaging  and  robotic  kinematic 
models  is  accomplished  using  a  recursive  least-squares  algo¬ 
rithm. 

While  the  methods  in  this  work  are  closely  related  to  the 
work  in  [1]-  [5],  the  specific  contributions  are  three-fold: 

1)  The  moving  target  problem  is  explicitly  addressed  and 
algorithms  are  derived  for  this  purpose.  Simulations 
verify  the  inappropriateness  of  the  static  controller  for 
the  moving  target  problem. 

2)  A  sound  theoretical  basis  is  established  for  uncalibrated 
vision-guided  control  algorithms. 

3)  The  algorithms  are  experimentally  verified,  and  model 
independent  vision-guided  robotic  control  and  tracking 
are  demonstrated  within  the  bounds  of  hardware  limita¬ 
tions. 
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Camera  1 


Fig.  4.  The  image  features  of  the  end  effector  and  the  target  as  seen  by 
Camera  1  are  shown.  The  average  steady  state  tracking  error  is  8.6  pixels. 


Time  (s) 

Fig.  5.  Error  in  pixels  for  Camera  0  as  robot  tracks  a  target  moving  on  a 
conveyor  belt.  The  arrows  denote  the  reversal  of  the  conveyor  belt  motion. 
Note  there  is  a  10-20  pixel  error  as  the  robot  overshoots  the  target  position. 
Tracking  is  regained  within  approximately  three  control  cycles. 
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